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SECTION I 



SUMMARY OF THE CALLING THE ROLL PROGRAM AND THE 

SEDL RESEARCH STUDY 



Calling the Roll 

Calling the Roll is a two-state education study circle^ program scheduled to 
take place during September through November of 1998 in Arkansas and 
Oklahoma. This project is a collaborative effort that has been ongoing since 
the fall of 1997, primarily through the efforts of the Southwest Educational 
Development Laboratory (SEDL), the Study Circles Resource Center (SCRC), 
Arkansas Friends for Better Schools (AFBS), and the League of Women 
Voters of Oklahoma (LWVO). These partners are working to plan, 
implement, and learn from study circles, a public engagement model that 
fosters dialogue among diverse individuals and groups. Two university- 
based partners — Center for Research on Teaching and Learning at the 
University of Arkansas at Little Rock and the University of Oklahoma, 
Norman Department of Sociology, representing AFBS and LWVO, 
respectively — are working with SEDL and SCRC to evaluate the program and 
examine its impact. 

SEDL's Participation in Calling the Roll 

Part of SEDL's organizational mission is to build and apply knowledge about 
how state education policy development can be improved. In 1997, SEDL staff 
identified public deliberation as a potentially important process for aiding and 
improving state policy work. SEDL staff, guided by experts in the field, chose 
the "study circle" model as an effective engagement tool to examine the 
process of public deliberation and its benefit to state policymakers. SEDL's 
primary interest in the Calling the Roll program is how policymaker 
participation in study circles affects the state education policy development 
process and the lessons to be learned from implementing a state-wide study 
circle program that includes policymakers. 

The two state partners (AFBS and LWVO) will provide the bulk of 
coordination and lead the implementation of the state-wide study circles. 
Because the benefit of study circles to the policymaking process is of key 
importance to SEDL, staff will work with these organizations to secure the 
participation of state-level policymakers in the program. SEDL staff will also 
work closely with SCRC to develop materials for study circle sessions. 



* Special terms related to the program and research project are defined in the glossary, which 
appears at the end of this document. 



including discussion materials for session three of the study circles and an 
education resource guide for each program state. 

To gather information related to its interest, SEDL will begin work on a 
multi-phase research study of Calling the Roll in June of 1998 and will 
continue this study beyond the 1999 legislative sessions in Arkansas and 
Oklahoma. Key features of the study are summarized below, and a detailed 
discussion of the research plan appears in Section IV of this document. 

CMllin^ the Roll Research Study 

SEDL's research study of Calling the Roll has two major goals: 1) explore how 
participation in study circles affects the education policymaking process, and 
2) learn about the process of implementing a state-wide program of study 
circles on education that includes state policymakers. SEDL staff will conduct 
an impact study of program effects on state oolicymaking and a process 
evaluation of the state-wide implementation of study circles in Arkansas and 
Oklahoma. These two complementary research approaches will rely on the 
following data collection methods: interviews, surveys, observation, and 
collection of written documents. In order to track the implementation 
process over time and to record short-, mid-, and long-term effects of the 
program, SEDL will collect data before, during, and after the study circles are 
implemented in the fall of 1998. Follow-up interviews and surveys also will 
be conducted to gauge the effect of study circles on state education policy 
activities during the 1999 legislative session. Analyses will be performed on 
qualitative data using a coding method developed by Strauss and Corbin 
(1990); frequency and central tendency calculations will be performed on 
quantitative research data. 

Document Oreanization 

The purpose of this document is to give a brief overview of the Calling the 
Roll program and to present the overall plan for the SEDL research study of 
Calling the Roll. SEDL will use this as a working document for SEDL staff, 
program partners, and external peer reviewers. 

Following this summary are four sections organized as follows: Section II 
presents background information on public engagement and the 
policymaking process; Section HI describes the state-wide implementation of 
the Calling the Roll program in Arkansas and Oklahoma; Section IV presents 
the SEDL research study; and Section V provides a brief overview of the 
collaborative research effort between SEDL, SCRC, UALR, and UON. A 
bibliography, a glossary, and a list of acronyms appear at the end of the 
document for the reader's reference. 
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SECTION II 



PUBLIC ExNGAGEMENT AND THE POLICYMAKING PROCESS: 
THE PROBLEM AND ITS CONTEXT IN RESEARCH AND 

PRACTICE' 



Education Reform and Local Communities 



Across the nation, education problems are framed, solutions designed, and 
reforms initiated in an environment of continued concern about the quality 
of public education. National and state education legislation as well as local 
school policies are continually created and enacted to address a wide range of 
issues such as school choice, school finance, curriculum content, and 
achievement standards. 

Generally, public opinion polls and focus groups reveal that local 
communities are concerned with the state of public schools and with current 
reform efforts (Johnson & Immerwahr, 1994; Johnson, 1995; Elam, Rose, & 
Gallup, 1996). Through these types of information-gathering processes, 
communities and individuals have expressed both a low level of confidence 
in public school quality and dissatisfaction with many recent attempts at 
school reforms. At the same time, however, these same individuals confess a 
low level of personal involvement with their local schools and infrequent 
participation in elections and school activities. These findings point to a wide 
gap between local communities and the process of education reform. A 
number of explanations exist for the disconnection between individuals and 
education policy, including public apathy, distrust in the policymaking 
process, a perceived lack of involvement opportunities, cultural barriers, and 
ineffective communication processes that engage local communities. 

Public Engagement and the Policymaking Process 

The growing tension between public dissatisfaction with education reform 
directions and powerlessness to effectively take part in guiding or setting 
these directions has important implications for those people who establish 
much of the context for local district and school reform — state policymakers. 
Legislators, state board of education members, and chief state school officers 
are faced with declining public trust in their decisions and a sometimes 
startling lack of public support for implementing the education reform 
policies they set into motion (Mutchler, 1993). 



^ A full discussion of public engagement and the policymaking process will be addressed in a 
forthcoming document that will present a literature review and conceptual analysis of 
deliberative dialogue and the policymaking process. That document will be incorporated in 
the results report of this research. 



Efforts by policymakers to gain greater certainty of constituents' positions and 
preferences relative to public education typically depend on time-honored 
methods of communication and information gathering. Policymakers rely on 
staff surveys of constituents, polls, information gathered from expert and 
non-expert interest groups, hearings, and informal communications with 
constituents. Unfortunately, these sources of information may neither 
accurately reflect majority constituent perceptions nor reveal certain minority 
perceptions. Moreover, they cannot reflect the opinion of all constituents 
who must accept and support education reform set in motion by state policy, 
if such reform is to be successfully implemented and sustained (Yankelovich, 
1991). 



Delihorative Dialogue — A Public Engagement Tool for the 

Policymaking Process 

These tensions call for an examination of why and how state policymakers 
might interact with the public in some fundamentally different ways as they 
gather information, debate, and ultimately create the state policy context for 
local education reform in our nation of increasingly diverse communities. In 
order to focus its policy-relevant research project on a specific public 
engagement model and to identify important uses of interactive processes for 
policymaking, SEDL solicited input from regional policy analysts during an 
October 1996 "Networkshop," from experts in public engagement at a 
February 1997 roundtable, and through individual interviews with 
policymakers in the spring of 1998. 

As a result of these information-gathering activities, SEDL identified 
"deliberative dialogue" as a public engagement tool of potential importance 
for state policymakers. Deliberative dialogue has a long history in the United 
States, beginning in the Chautauqua adult education movement of the late 
19th century. Across the nation, deliberative dialogue is now being 
implemented by community service organizations, interfaith religious 
groups, public agencies, and others to help the public gain a better 
understanding of local problems and build commitment toward resolving 
them. 

Input from these meetings and interviews also contributed to the following 
conclusions regarding deliberative dialogue and education policy^: 

• Deliberative dialogue is an inclusive, democratic public engagement 
method that asserts the responsibility of individual stakeholders 
and gives them a role in setting the social or political agenda by 



^ Detailed results of these initial inquiries will be available in a separate document entitled 
"Results from Interviews of Policymaker Users of Deliberative Dialogue" (currently in draft 
form). 



involving them in weighing multiple perspectives, choices, and 
consequences. 

• Output from deliberative dialogue appears to include better 
informed (although not necessarily consensus) opinions, clearer 
definitions of persistent problems, more coherent perceptions of the 
range of solutions and their consequences, and a sense of public 
priorities. 

• Promising uses of dialogue by state education policymakers might 
include a way to better manage tough policy decisions, gauge the 
level of public concern about particular problems, test ideas, discern 
the majority point of view, redefine the social compact around 
public education goals, and build "social capital" for implementing 
decisions. 

Study Circle Model for Encouraging Deliberative Dialogue 

About Education 



There are a number of interactive processes that are being used across the 
nation to accomplish these kinds of public engagement goals in the context of 
improving education. National organizations dedicated to education (among 
them the Annenberg Institute for School Reform, Education Commission of 
the States, the Institute for Educational Leadership, the National Parents and 
Teachers Association, and Phi Delta Kappa) are applying and studying 
dialogue strategies, most of which are based largely on two existing models 
refined and facilitated by private, non-profit foundations: study circles (Study 
Circles Resource Center, Pomfret, Connecticut) and National Issues Forums 
(Kettering Foundation, Dayton, Ohio). 

The study circle process is judged by SEDL to be particularly suited to engaging 
state policymakers and the public in dialogues about education for two 
reasons. First, at the study circle level itself, community members and their 
public officials interact about education in ways that differ from traditional 
public polls, legislative hearings, strategic planning sessions, school board 
meetings, and other settings. The process is a semi-structured, multi-step 
approach to engaging people over time in small group discussions. The study 
circle is facilitated by a community member trained to lead civil, thoughtful 
dialogue. The study guide on education developed by the Study Circles 
Resource Center is used to direct participant dialogue through four or more 
meeting sessions — progressing from discussion of personal experiences about 
education, to deliberation about different perspectives on education issues, 
and finally to consideration of possible actions that might solve problems 
identified by study circle participants. 



Second, the study circle process is increasingly being implemented as a 
community-wide model capable of engaging broader and more diverse 
segments of the public and providing a structure for identifying joint action 
to solve community problems. During the last five years, local groups 
throughout the nation have coordinated and convened community-wide 
study circle programs that often result in community action, including local 
policy change (SCRC, 1997a). This expansion of deliberative dialogue to 
include more constituents and consider policy-relevant action has potential 
impact not only for local communities, as they work to improve their public 
schools, but also for state policymakers as they work to improve their state 
education policymaking. 
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SECTION III 



DESCRIPTION OF THE FIELD PROGRAM, CALLING THE ROLL 



This section provides an overview of the Calling the Roll program, including 
a summary of major events, description of the interests that each partner 
brings to the program, and a program timeline. Note that this section is 
meant to provide a context for the SEDL research plan that begins in detail in 
Section IV. 



Program Summary 

As stated previously. Calling the Roll (CR) involves the implementation of 
the study circle model in communities throughout the states of Arkansas and 
Oklahoma during the fall of 1998. CR planning and collaboration among 
SEDL, the Study Circles Resource Center (SCRC), and two state partner 
organizations (Arkansas Friends for Better Schools and the League of Women 
Voters of Oklahoma) have been ongoing since the fall of 1997. The interests of 
each program partner differ, ranging from developing a public engagement 
process useful for state policymakers to increasing civic participation in 
education, to encouraging discussion on education issues. All partners also 
share an interest in studying the process and impacts associated with the 
Calling the Roll program. 

Each state partner organization will sponsor the effort in its own state, 
develop its own coalitions, build local networks, coordinate the CR program, 
and provide support and information for research activities. SCRC will 
provide technical assistance in facilitating community-wide study circle 
programs, ranging from advice on organizing strategies to help on 
communications and facilitator training. SCRC will also conduct a locally- 
focused process evaluation. SEDL will assist in implementing the study circle 
program in both states by informing and involving state policymakers in the 
program, by developing program discussion materials (including state-specific 
education resource guides), and by providing seed money to each of the state 
coordinating organizations. Most importantly, as discussed in Section IV of 
this document, SEDL will investigate the study circle model, explore impacts 
on the education policymaking process, and develop products to disseminate 
learnings. 

Table 3.1 on the next page presents an overview of the responsibilities and 
roles of each of the program partners. 



Table 3.1: Calling the Roll Partner Responsibilities and Roles 



Partner Responsibility 


SEDL 


SCRC 


AFBS 


LWVO 


Coordinate at multi-state level 


X 


X 






Coordinate at state level 






X 


X 


Coordinate at community level 






X 


X 


Engage policymakers 


X 




X 


X 


Provide resources 


X 


X 


X 


X 


Provide technical assistance 




X 






Conduct or support research study on CR^ 


X 


X 


X 


X 


Publish report on research results^ 


X 


X 


X 


X 


Develop products for dissemination^ 
^Includes post-program activities 


X 


X 







The two state partner organizations (AFBS and LWVO) are contracted to 
coordinate the CR program in ten or more communities in their respective 
states. Community sponsors will coordinate a kick-off forum, a series of four 
study circle sessions in one or more sites, and a community-wide action 
forum in their respective communities. Study circles will be open to all who 
wish to participate. AFBS and LWVO will implement programs in both 
urban and rural communities and will emphasize recruitment of study circle 
participants who reflect the diversity of their state's populace. 

Through letters and personal communications, SEDL staff will recruit up to 
ten policymakers (defined as legislators and other key state-level education 
decisionmakers) in each state to participate in study circles (preferably one 
policymaker per community). An additional three policymakers among 
those who are unable to attend the study circle sessions will be recruited in 
each state to receive reports generated as a result of the study circles. SEDL 
also will invite elected legislative aides, governors' education liaisons, and 
policy analysts from the state departments of education to participate in or 
observe study circles. 

As a result of the collaborative planning process, program partners decided to 
adopt a study circle discussion guide that was used with previous study circle 
programs on education. This discussion guide. Education: How Can Schools 
and Communities Work Together to Meet the Challenge, was developed by 
SCRC to facilitate discussion around education issues that are important at 
the community level. However, to ensure that a portion of the study circle 
discussion guide covers an issue relevant to state education policy 
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development, the partners will work to develop discussion materials on 
accountability to replace the existing study circle session three guide. 

Interests of Program Partners 

Each CR partner has specific interests in the program that are unique to the 
needs of its state and/or organization as discussed in the text below and 
summarized in Table 3.2. 

SEDL's overall mission is to improve education within the Southwestern 
Region, and its specific interest in the CR program is the possibility of 
enhancing the policymaking process through dialogue among policymakers 
and the public on education issues. As elaborated in the description of the 
SEDL research study (beginning in Section IV), SEDL plans to use the 
opportunities that CR presents to learn about program impacts on the state 
education policymaking process and to examine strategies important to 
implementing and involving state policymakers in state-wide study circles on 
education. 

SCRC is interested in learning what planning and implementation needs 
arise from scaling up the community-wide study circle program model to the 
state-wide and multi-state levels. SCRC is also interested in learning about 
involving policymakers in the process and learning more about the general 
effectiveness of study circles. 

Arkansas Friends for Better Schools has a specific interest in implementing 
(and eventually institutionalizing) the study circle model to enhance public 
knowledge about education and increase public support and participation in 
public schools. The League of Women Voters in Oklahoma would like to 
inform individuals on education issues and increase participation in civic 
matters related to education, including increased interaction between public 
officials and constituents. 



Table 3.2; Calling the Roll Program Partners' Interests 



SEDL 1. Increase collaboration with state and local entities whose mission is to 

improve public education. 

2. Enhance the connection between policymakers and the public for the 
purpose of improving education policy. 

3. Learn from the implementation of study circles on education that 
include policymakers, documenting impacts on state education 
policymaking and gaining insights on successful implementation 
strategies. 



SCRC 



AFBS 



LWVO 



1 . Learn how to adapt the commimity-wide SC model and SCRC 
technical assistance to support state-wide and multi-state study circle 
programs. 

2. Learn more about the effectiveness of study circle programs. 

3. Study the process of including policymakers in study circles. 

1 . Increase community involvement in and support of public schools. 

2. Implement community-wide study circles on education and 
institutionalize the study circle model as a problem-solving strategy for 
communities. 

3 . Enhance public knowledge, understanding, and attitudes about public 
education. 

1 . Educate the public on education issues. 

2. Enhance civic participation, particularly with respect to schools and 
other educational opportunities. 

3 . Increase direct involvement of public officials with constituents. 



At present, each partner is conducting a separate research study based on one 
or more of its specific interests in participating in the program. The result will 
be an interrelated and mutually supporting set of studies that "capture the 
whole story" of the 1998 state-wide study circle program on education in 
Arkansas and Oklahoma. The research plan for SEDL's study of Calling the 
Roll appears in Section IV, beginning on page 12. Further details of the 
collaborative research are presented in Section V of this document. 

Calling the Roll Program and Post-Program Timeline 

Planning and development activities for Calling the Roll began in the fall of 
1997, and program coordination and implementation will culminate with the 
study circle sessions scheduled to take place between September 1 and 
November 30, 1998. Although local study circle programs are expected to 
finish by December 1998, research and dissemination activities in the 
Southwestern Region will be ongoing through 1999. A timeline for Calling 
the Roll and related post-program activities appears on the next page, 
highlighting major tasks by category. A timeline specifying the data collection 
activities to be conducted by SEDL is presented and discussed in Section IV. 



State-wide Study Circle Program on Education 

Program Timeline 
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SECTION IV 



DESCRIPTION OF THE SEDL RESEARCH STUDY 
OF CALLING THE ROLL 



Introduction 



As briefly discussed in the previous section, one of SEDL's interests in the 
Calling the Roll program is the opportunity that the program presents for 
learning about the study circle process and how it might affect state 
policymaking. Towards this end, SEDL has planned a research study of the 
CR program that will examine both the impact of such a program on state 
education policymaking and the process of implementing a state-wide 
program of study circles on education that include state policymakers. 
Research results will be used to inform a number of different products, 
including a final report on research findings, short informational reports for 
use by policymakers and other interested audiences, and an informational 
video. 

A brief schematic appears on the next page that summarizes major elements 
of the SEDL research study, including research goals, key questions, design 
approaches, data sources, and expected products. As the graphic shows, SEDL 
will pursue two primary research goals and follow two distinct research 
approaches. The first goal and corresponding approach will allow SEDL to 
uncover policy-relevant impacts of the program. The second goal and 
corresponding approach will allow SEDL to examine the process of CR 
implementation. Each research goal has been divided into a set of key 
questions that will be explored using a variety of data collection methods. 

Immediately following the graphic is a detailed description of each of the 
elements displayed which elaborates on the scope of the research. While the 
basic structure of this research likely will remain unaltered, changes in 
coordination efforts or conditions in each state may create the need for 
modifications ot the procedures. 
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OVERVIEW OF THE SEDL RESEARCH STUDY 






Gcals for SEDL Research Study^ 



1) To explore how state policymaker 2) To learn about the process of 

participation in study circles affects the implementing a state-wide program of 

education policymaking process. study circles on education that include 

state policymakers. 



I 



Key Research Questions 






Impact Study 

1 . How do study circles on education provide 
state policymakers and their constituents 
with useful information? 

2. What is the impact of study circles on state 
policymakers’ relationships with constituents, 
as related to education? 

3. What is the impact of study circles on state 
education policymaking activities and 
relevant civic participation? 



Process Evaluation 

1 . How is a state-wide program of study circles 
on education planned and implemented? 

2. What strategies are used to involve state 
policymakers in study circles on education? 

3. What are the costs of implementing and 
involving state policymakers in a state-wide 
program of study circles? 



Research Approach 



Non-experimental, exploratory, 
descriptive study of impact of study circles 



Evaluation of program implementation 
process 



Data Sources 



Interviews with policymakers 
Interviews with state coordinators and regional 
implementation staff 

Interviews with selected expert public participants 
Interviews with selected non-expert public participants 
Surveys of state policymakers 
Surveys of program participants 
Observation of select study circle sessions 
Review of documents 

Products 



Technical report on results of research study 

Guidebook for policymakers, agencies, and potential organizational sponsors of study circles 
Informational reports for policymakers (policy briefs) 

Informational video documenting the Calling the Ro// implementation process and results 




®The goals for this research study were developed based on SEDL’s interest in learning from the implementation of state- wide programs of 
study circles on education that include policymakers, stated in Section III, page 10 of this document. 
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Research Goals 



As identified in Section III of this document, one of SEDL's major interests i n 
the CR program is as follows: 

Leavnfrom the iTnplernentution of study circles on education that include policymakers, 
documenting impacts on state education policymaking and gaining insights on 
successful implementation strategies. 

This interest provides the basis for development of this research study of 
Calling the Roll and also guides the two major goals for the study. First, 
researchers will explore the impact of policymaker participation in study 
circles on the state education policy development process. Second, SEDL will 
examine the process of implementing a state-wide program of study circles on 
education that includes policymaker participation. The primary goals that 
will guide SEDL staff throughout the research study are a^ follows; 

Goal One: To explore how state policymaker participation in study circles 
affects the education policymaking process. 

Goal Two: To learn about the process of implementing a state-wide program of 
study circles on education that includes state policymakers. 

In developing the research design, SEDL recognized the need to pursue two 
distinct research approaches to fully address the research goals listed above. 
The research approach appropriate to goal one is an exploratory descriptive 
study of program impacts, and the research approach appropriate to goal two 
is a process evaluation. These two approaches are described further in the 
two sections below. 

Goal One: Key Questions and Research Approach 

Goal One Questions and Themes 

The first goal of SEDL's research study is to explore the impact of study circles 
on the state education policymaking process. The intent is to uncover 
evidence as to whether study circles provide a qualitatively different way for 
state policymakers and constituents to interact for the purpose of addressing 
public education and education policy issues. Three key questions will be used 
to frame goal one research; 

1. How do study circles on education provide state policymakers and their 
constituents with useful information? 

2. What is the impact of study circles on state policymakers' relationships with 
constituents, as related to education? 
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3. What is the impact of study circles on state education policymaking activities 
and relevant civic participation? 

The above questions make use of three broad outcome categories associated 
with state policymaker participation in study circles: informational, 

relational, and civic action.® Informational outcomes include achieving 
access to different information sources, multiple viewpoints, and levels of 
expertise. Relational outcomes include changes in perceptions of or 
interaction between individuals or groups. Civic action outcomes include 
both changes in the state policymaking process and participation in civic 
activities. 

Table 4.1 below illustrates the key questions for goal one and corresponding 
themes to be explored. 



Table 4.1: Goal One Key Questions And Corresponding Themes 





GOAL ONE KEY QUESTIONS 




1. How do study circles on 


2. What is the impact of 


3. What is the impact of 




education provide state 


study circles on state 


study circles on state 




policymakers and their 


policymakers' 


education 




constituents with useful 


relationships with 


policymaking 




information? 


constituents, as related 
to education? 


activities and relevant 
civic participation? 


C/5 


• Identification of issues 


• Interaction in a safe, non- 


• Increased willingness to 


S w 


important to individuals 


confrontational 


take risks in developing 


and communities 


environment 


new policies 


• Understanding of common 


• Increased willingness to 


• Increased participation in 


0 p 


ground and of the public's 


listen to others rather than 


or organizing of civic 




willingness to weather 


speak 


activities 


< < 


trade-offs resulting from 


• Forging of previously 


• Irmrovement of dvic 
infrastructure and 


0 K 


decisions 


unidentified alliances 


0 w 


• Access to multiple layers of 


(common ground) 


networks 


m 

0 


experience and expertise 
• Exposure to diverse 
viewpoints 


• Changes in perceptions of 
other mdividuals or groups 





^ SEDL staff have identified these three categories based on information gathered during the 
design phase of this research project. For a full discussion of these outcome categories, see 
"Results from Interviews of Policymaker Users of Deliberative Dialogue," currently in draft 
form. 



Goal One Research Approach 



In designing the research approach for goal one of this study, SEDL staff 
decided to adopt a design that would help to obtain the most complete picture 
of the complex phenomenon under study. Given the fact that no research 
findings currently exist on the impact of study circles on policymaking and 
given the unique program elements of Calling the Roll (participation of 
policymakers and state-wide implementation), an exploratory descriptive 
research approach that relies predominately on qualitative data will be used. 
This approach, referred to in this document as an "impact study," emphasizes 
an in-depth description of how the program affects the education 
policymaking process, using interviews, observation, and the inductive 
abilities of the researchers as primary sources of data. 

The impact study begins with general themes to be explored, in contrast to a 
narrow set of expected outcomes or the more detailed sub-themes that are 
being used to guide the goal two approach, outlined below. As is clear from 
Table 4.1, the general themes developed by SEDL are intentionally broad so 
that they will provide fundamental foci for the study but will also allow 
freedom to explore additional related phenomena that might emerge in the 
course of the study. 



Goal Two: Key Questions and Research Approach 

Goal Two Questions and Themes 

The second research goal is concerned with documenting the Calling the Roll 
program and examining the implementation process, including planning, 
coordinating, and conducting the study circle sessions in the fall of 1998. 
Researchers will identify successful strategies and barriers related to the 
participation of policymakers in the program, and document the resources, 
planning, and coordination necessary to implement a state-wide study circle 
program on education. Three key questions will be asked in pursuit of goal 
two: 

1. How is a state-wide program of study circles on education planned 
and implemented? 

Study circles have traditionally been used at the small group or local level 
and may or may not involve an education topic. Since coordination on 
community-wide and state-wide levels is critical to the program's 
relevance to state education policymakers, the model's applicability to 
organizing and implementing education discussions across a state will be 
an important focus of investigation. 
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2. What strategies are used to involve state policymakers in study 
circles on education? 

State education policymakers as well as their constituents will be 
targeted for participation in study circles. Modifications in program 
planning or implementation that facilitate the involvement of 
policymakers in the program will be examined. 

3. What are the costs of implementing and involving state 
policymakers in a state-wide program of study circles? 

The success of a state-wide program depends on the provision of 
resources, either in-kind or monetary, for coordination, publicity, 
facilities, staffing, materials, travel expenses, and office expenses. The 
resources necessary for implementing and sustaining Calling the Roll will 
be tracked and analyzed by SEDL staff. 

As the three key questions reveal, SEDL researchers will look at the study 
circle program implementation process in terms of its relevance for the 
successful participation of policymakers and their constituents. Researchers 
will look at a wide variety of elements necessary for successful 
implementation of study circles on education that occur at the state-wide 
level. Since program implementers will follow the basic steps of organizing a 
community-wide study circle program as described by the Study Circles 
Resource Center, SEDL will also track these steps as evaluation themes. The 
outline below provides a list of themes that will be examined by the SEDL 
research study. Table 4.2 displays these themes along with detailed sub- 
themes, their relation to the key questions of goal two, and key data sources. 

Coordination (state and local): In order to hold a successful state-wide study 
circle program with multiple local sites and many participating groups, 
coordination is necessary both at the state and local levels. SEDL will track 
the state coordination of Calling the Roll — documenting how state 
coordinators work to develop local coalitions, incorporate the participation of 
policymakers and constituents, and develop resources for the project. 

Coalition building: Study circles are meant to be highly participatory, and the 
strength of the study circle model is that diverse groups and multiple sectors 
of the community are drawn into discussions. Policymaker interaction with 
constituents of all levels in the community is also a key feature of Calling the 
Roll. Coalition building at the state level and the relationship between state 
coordination and local coalitions are key foci of the study. 

Selecting andfor writing session materials: Study circle discussions are guided 
by a set of written participant materials. A previously existing discussion 
guide for education study circles. Education: How Can Schools and 

Communities Work Together to Meet the Challenge? , will be modified for 
the CR program to enable policymakers and constituents to discuss a state 
policy issue. SCRC and SEDL will modify session three from this discussion 



guide with input from CR partners and guidance from outside experts. SEDL 
also will produce a resource guide of education data and information specific 
to each state. The process of such materials development as well as the use of 
the materials in study circles will be documented for this research. 

Recruitment (facilitators, policymakers, and constituents): Study circle 
program implementation will involve recruitment at a number of different 
levels. As described in Section III of this document, SEDL will work to help 
recruit policymakers to participate in the CR program. Also, state and local 
coordinators will recruit other participants, with emphasis on diversity in 
order to bring all levels of the policymaker's constituency to the discussions 
on education. Recruitment strategies, barriers, and effects will be a major 
theme of the process evaluation. 

Media coverage and other publicity of events: In order to reach a wide 
audience and ensure broad participation, publicity will be a key feature of 
program implementation. Through interviews of and logs from state 
coordinators, SEDL will document the use of media and other ways of 
generating community interest in the program and also will track related 
news coverage. The use of different media and publicity strategies by state 
coordinators will be observed, and coordinators will be asked to provide their 
perceptions of the success of their strategies. 

Study circle sessions: Although Calling the Roll will involve study circles that 
are uniformly planned to use identical discussion materials and study topics, 
each circle will be unique in terms of the make up of the circles and the 
interests that individuals bring into the process. This study will examine the 
dynamics of those study circle sessions that include policymakers. 

Sustainability of state-wide study circle programs on education: Although this 
last theme is not within the list of basic steps developed by SCRC, SEDL will 
explore the appropriateness and feasibility of longer term or subsequent 
interaction between policymakers and their constituents through study 
circles. Researchers will also identify elements of Calling the Roll that might 
result in longer-term operation of study circles on education in the program 
states. 
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TABLE 4-2: Goal Two Key Questions, Corresponding Themes, and Sub-themes 



GOAL TWO 
THEMES 


GOAL TWO KEY QUESTIONS 


DATA 

SOURCES 


1. How is a state- 
loide program of 
study circles on 
education planned 
and implemented? 


2. What strategies 
are used to involve 
state policymakers 
in study circles on 
education? 


3. What are the costs 
of implementing 
and involving 
state policymakers 
in a state-wide 
program of study 
circles? 


Coordination 


• State and local 
coalition building 

• Local site selection 

• Coordination barriers 


• State coordinator role 
and SEDL role in 
policymaker 
involvement 


• Staffing/ funding 

• Essential resources 

• Volunteer capacity 

• Education system 
contribution 


• Interviews with 
program staff 

• Review of coordinator 
logs 

• Policymaker interviews 


Coalition 

Building 


• Role of coalition 

• Diversity of sponsors 
and participants 

• Barriers/successes 


• Role of elected/ 
school officials, and 
policymakers in state 
and local coalitions 


• Staffing/funding 

• Essential resources 

• Volunteer capacity 

• Education system 
contribution 


• Interviews with 
program staff 

• Review of coordinator 
logs 

• Policymaker interviews 


Selecting/ 

Writing 

Materials 


• Development and 
use of new materials 

• Presentation of issues 


• Utility of materials to 
policymakers 


• Costs of development 
and distribution 

• Availability and use 
of expert Imowledge 


• Interviews with 
progTcun staff 

• Review of coordinator 
logs 

• Policymciker interviews 


Recruitment 


• Barriers and 
effective strategies 

• Role of publicity 

• Effect of education 
topic 


• Barriers and effective 
strategies 

• Role of publicity 

• Effect of policymaker 
participation 

• Effect of education 
topic 


• Staffing/funding 

• Essential resources 

• Volunteer capacity 

• Education system 
contribution 


• Interviews with 
progTcun staff 

• Review of coordinator 
logs 

• Policymciker cmd 
participant surveys 

• Document review 

• Policymciker cmd 
participant interviews 


Media and 
Publicity 


• CR-initiated vs. self- 
generated media 
coverage 

• Role of media, kick 
off, etc. 

• Other uses of media 
(internet, flyers, 
posters, etc.) 


• Effect of publicity on 
policymakers 

• Role of policymakers 
in generating 
publicity 

• Effect of policymaker 
appearance at kick 
off or action forum 


• Costs of media 
coverage or publicity 

• Staffing necessary for 
effective publicity 

• Volunteer capacity 


• Interviews with 
progTcim staff 

• Review of coordinator 
logs 

• Policymciker cmd 
participcmt surveys 

• Document review 

• Policymciker and 
participant interviews 


Study Circle 
Sessions 


• Consistency of 
session witb study 
guide 

• Education concerns 

• Effect of current 
events 

• Problems/Successes 


• Role of policymaker 

• Levels of policymaker 
participation 

• Ed. concerns of 
policymakers 

• Problems /Successes 


• Staffing/ funding to 
support sessions 

• Changes in facilitator 
training to 
incorporate 
policymakers 

• Facilities cost 

• Volunteer capacity 


• Interviews with 
progTcim staff 

• Review of coordinator 
logs 

• Policymciker and 
participant surveys 

• Document review 

• Policymaker cmd 
participant interviews 


Sustainability 
of State-wide 
Programs 


• Identification of 
practices that foster 
sustainability 

• Role of education 
topic in sustainability 

• Identification of 
follow-up efforts 
and new planning 
activities 


• Policymaker interest 
in participating in 
future programs 

• State-level 
organizations' 
interest in sponsoring 
programs 


• Staffing/ funding to 
support ongoing 
programs 

• Volunteer capacity 

• Sustained technical 
support 


• Interviews with 
program staff 

• Policymaker cmd 
participant surveys 

• Policymaker and 
participant interviews 
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Goal Two Research Approach 



SEDL will document and describe the process of implementing and managing 
a state-wide program of study circles on education to address the questions 
posed for goal two of this study. An evaluation of the process will allow 
researchers to examine and assess the design, planning, and implementation 
of Calling the Roll for the purpose of gaining information about the study 
circle process and the feasibility of including policymakers in a state-wide 
program. Since this evaluation will document the implementation process, 
SEDL has designated the goal two research approach as a "process 
evaluation."^ The results of the process evaluation will be used as contextual 
background information for the impact study that was described above, and 
will provide data for products developed by SEDL to assist other organizations 
in implementing future state-wide programs of study circles on education. 



Goal One and Goal Two Data Sources 



As discussed in the preceding pages, the SEDL research study will examine 
both the impact of the CR program and the implementation process using 
two separate research approaches. An impact study will explore the effects of 
study circles on the policymaking process (goal one), and a process evaluation 
will focus on describing the process of implementing and managing a state- 
wide program of study circles on education that includes policymakers and 
their constituents (goal two). Both approaches will rely on the same data 
sources as listed in the table on the next page. The table also presents the 
relative importance each data source has for the two research goals. 




’’ Rossi and Freeman (1993) discuss process evaluations and program monitoring in detail. Goal 
two key questions loosely follow the three elements of process studies presented by Rossi and 
Freeman: 1) service delivery and program design, 2) ability of intervention to reach the target 
population, and 3) resources expended for program implementation. 
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Table 4.3: Primary and Secondary Data Sources for Research Goals 



GOAL ONE AND GOAL TWO 
DATA SOURCES 


GOAL ONE 


GOAL TWO 


Primary 

source 


Secondary 

source 


Primary 

source 


Secondary 

source 


Interviews with policymakers 


X 






X 


Interviews with state coordinators and 
regional implementation staff (AFBS, 
LWVO, SCRC, SEDL staff) 




X 


X 




Interviews with selected expert participants 


X 






X 


Interviews with selected non-expert 
participants 


X 






X 


Surveys of policymakers 


X 




X 




Surveys of participants 


X 




X 




Observation of select study circle sessions 


X 




X 




Review of documents (program reports, 
coordinator monthly logs, local and state 
media reports, legislative reports) 


X 




X 





Description of Data Sources and Collection Procedures 

The data sources in Table 4.3 represent an appropriate mix for this study. A 
description of each data source and the actual procedures necessary to collect 
them are presented below. Site and sample selection criteria appear 
immediately after. 

Interviews 

Guided interviews with individuals are used to document both program 
impacts (goal one) and the implementation process (goal two). Key 
informants who will be interviewed include state policymakers, state 
coordinators (AFBS and LWVO), regional implementation staff (SCRC and 
SEDL), expert study circle participants (educators, administrators, and others 
associated with the education profession) and non-expert participants 
(community members). 

SEDL will develop written guides that will help structure the interviews. 
Face-to-face or phone interviews will be conducted by SEDL researchers, with 
limited assistance from hired interviewers. The interview guides will allow 
for a semi-structured question-and-answer session and will be developed for 
each of the populations of interest. All interviews will be audiotaped. At least 
50 percent of each round of interviews will be fully transcribed. Any 
remaining interviews will be selectively transcribed (transcriptions of 
particularly relevant themes). 



Interviews will be conducted periodically with state coordinators (2 
individuals), SCRC technical assistance staff (2), and SEDL staff (1) who are 
involved in the implementation process (see Table 4.5). Since interviews 
with these state coordinators and regional implementation staff will be used 
primarily to inform the sub-themes developed for goal two of the SEDL 
research study, the interview guide developed for these interviews will 
contain an extensive list of detailed questions. Interviewers will be instructed 
to pursue all questions contained in the guide. Consequently, these 
interviews are expected to inform a predetermined set of issues, and 
interviewee responses will be more tightly bounded than in interviews with 
other key informants. 

For the state policymaker, expert public, and non-expert public interviews, 
SEDL will develop more loosely defined interview guides that are meant to 
allow interviewers and interviewees the freedom to pursue the general 
themes for goal one and uncover unexpected therries of significance. Also, as 
the study progresses, questions will be altered in order to explore constructs of 
interest that emerge from earlier data. This approach puts much of the 
responsibility on SEDL staff to prioritize and develop research themes 
throughout the study. In order to control the bias inherent in this approach, 
monitoring of the process will be performed as described starting on page 31. 

Key^informant interviews will be conducted with up to 20 state policymakers 
(ten from each state) who attend study circles in fall 1998, and six state 
policymakers who receive program reports only. Policymakers will be 
recruited and interviewed on a volunteer basis. Baseline interviews will be 
completed with the policymakers prior to their involvement with the 
program. Policymakers will be interviewed again following their 
participation in the study circles (or following their receipt of information 
generated by the study circles, if they do not participate). Policymakers will be 
interviewed a third time following the 1999 legislative session to assess the 
impact of the CR program on their personal decisionmaking activities related 
to education policy development. Follow-up interviews will be performed 
with a sub-sample of these policymakers (five from each state) approximately 
three months after these post-session interviews in order to assess longer- 
term impacts. 

A limited set of interviews will be conducted with select members of the 
public who participated in study circles. These interviews will help track 
impacts related to interactions and information sharing between 
policymakers and their constituents, as well as civic action that might result 
from such interactions. During the study circle sessions, all participants will 
be invited to volunteer to be interviewed by providing their discussion 
facilitators with contact information for use by SEDL. Interviewees will be 
selectively sampled from those who participated in study circles in which a 
state policymaker was present. Initial interviews with ten members of the 
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expert public and ten members of the non-expert public in each state will be 
conducted after the study circle program has concluded. These post-program 
interviews will require the individuals to reflect retrospectively on their 
attitudes and behavior prior to, during, and after their involvement with 
study circles. Follow-up interviews will be conducted with a sub-sample of 
these same individuals approximately eight months after the end of the CR 
program. This sample (five members of the expert public and five members 
of the non-expert public per state) will be selected based on their previous 
interview responses. 

Surveys 

A variety of survey instruments will be developed by SEDL staff to collect 
more quantitative information than can be collected using interviews. 
Surveys will function both to facilitate the collection of basic demographic 
and attitude information from a larger sample of individuals than would be 
possible with interviews and to corroborate information obtained through 
interviews. Surveys will be administered to all policymakers who represent 
CR communities as well as to all participants (both policymaker and 
constituent participants) who attended Calling the Roll study circles. 

Policymaker Surveys. A pre-program survey will be sent via mail to all 
legislators who represent the CR communities in both states, whether or not 
they plan to participate in the program. This survey will be administered 
prior to the start of the CR program in either state, to provide important 
baseline data that will inform SEDL's general knowledge about public 
engagement relative to the legislative policymaking process. A follow-up 
survey will be administered six to eight months following the conclusion of 
the program to these same legislators. Information gained from the pre- 
program and follow-up surveys of policymakers will be used for multiple 
purposes, including learning about public engagement and education 
policymaking, gauging policymaker interest in the study circle model, and 
measuring long term impacts of the study circle process on their work. 

Participant Surveys (policymakers and constituents). SEDL will work with the 
other CR research partners to survey all participants who attend CR study 
circles. Since each of the program partners is interested in one or more 
aspects of the impact of study circles on participants, pre-program and post- 
program surveys will be incorporated into the actual implementation of 
study circles. Researchers from SEDL, SCRC, UALR, and UON will work 
collaboratively to develop and distribute these surveys that will be 
administered by study circle facilitators with the assistance of program 
coordinators. Approximately 1,000 study circle participants (policymakers and 
community members) are estimated to participate in the pre-program and 
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post-program surveys.® While the surveys will document a wide range of 
topics (satisfaction with the process, critical incidents that occurred during the 
sessions, changes in opinion and behavior as a result of participation, and 
basic demographic information), SEDL's interest in this survey focuses on 
participants as the constituents that policymakers represent. Specifically, 
SEDL is interested in the effectiveness of the study circles in helping 
policymakers and constituents interact, the information that is shared among 
stakeholders and between stakeholders and policymakers in the dialogue 
process, and civic participation patterns that might be affected by dialogue 
with state policymakers. 

In addition to the pre-program and post-program participant survey, SEDL 
will work with program partners to administer an in-program survey to 
approximately 200 participants who attend study circles that include 
policymakers.^ Facilitators will be asked to hand out and collect this survey at 
the end of the third study circle session using guidelines provided by CR 
researchers. The in-program survey will be used to document participant 
satisfaction with the process, to record important discussion topics that 
emerge from this session which is devoted to a policy-relevant issue, and to 
find out perceptions regarding the participation of a state education 
policymaker in the study circle. 

The majority of survey questions will use a forced choice format or rating 
scale. Selective use will be made of open-ended questions. All surveys will be 
self-administered. A determination will be made as to whether it will be 
necessary to provide surveys for community members in both English and 
Spanish. Also, provisions will be made to enable individuals with lower 
reading levels to complete the survey. 

Observation 

SEDL researchers will use observation to inform both the impact study and 
the process evaluation. Trained observers (e.g., graduate students from local 
universities and state education policy analysts) will be assigned to observe up 
to six study circL^ that will be selected from the study circles that include state 
policymakers. SEDL researchers also will perform observations, both to gather 
process information and to gain first-hand familiarity with the range of 
interaction that occurs within study circles. Non-SEDL observers will receive 
instruction from SEDL staff to help sensitize them to the key issues under 
study. All observers will use a checklist to monitor the integrity with which 
the study circles are implemented and an observation guide to describe the 



* Based on an estimate of 50 participants in each of 20 program sites. 

’ Based on an estimate of 10 participants in each of the 20 study circles that contain a 
policymaker. 
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interactions that occur between participants and state policymakers during the 
process. 

Activity Logs 

Monthly activity logs will be collected from the two state coordinators to gain 
insights on the implementation process in a more immediate and tangible 
fashion than is possible through periodic interviews. State coordinators also 
will be asked to provide copies of related artifacts (memos, letters, meeting 
agendas and summaries, press releases, evidence of press coverage). 

The logs also will be used to stimulate memory during retrospective portions 
of the interviews with state coordinators and cue the interviewer to specific 
questions that should be asked. In effect, activity logs and associated artifacts 
will provide a different type of information on the process of implementation 
and also will serve to increase the dependability of the findings. 

Document Review 

Document collection and review will be used to provide additional 
information regarding the implementation of state-wide study circle 
programs for the process evaluation and to provide information on the 
educational context that might have a bearing on the analysis of impact data. 
Three major sources will be tapped for documentation: 

1. SEDL staff will collect information from state (Arkansas and Oklahoma) 
and national media (primarily newspapers) on issues that relate to education 
or the study circle process. SEDL wall use an on-line, searchable, full-text 
news service (Lexis Nexis) to collect this information. Staff will institute a 
protocol for document sorting and storage using both electronic and hard- 
copy cataloging systems. 

2. SEDL wall collect and review program reports that are generated by Calling 
the Roll recorders and coordinators. These reports wall provide further 
primary and secondary data on the program implementation process and 
local and state-level impacts. These "reports" may be in a formal written 
format synthesized by state coordinators, or they may be original summaries 
of community-level record keeping. 

3. In order to track major legislative issues on education in Arkansas and 
Oklahoma and the progress of 1999 education bills, SEDL will scan 
legislative updates made available by legislative staff in electronic or hard- 
copy format. 



Site and Sample Selection Criteria 



As the discussion of the data sources reveals, a large number of interviews, 
surveys, and observations will be conducted for the research study of Calling 
the Roll. For the reader's convenience. Table 4.4 summarizes the site and 
sample selection criteria to be used for each data source. 



Table 4.4: Site and Sample Selection Criteria for Interviews and Surveys to be 
Conducted for SEDL's Calling the Roll Research Study'® 



PROCEDURE 


SITE/SAMPLE SELECTION CRITERIA 


Coordinator Interviews 


All state coordinators, SCRC technical assistance staff, and SEDL 
Staff who assist with CR implementation will be interviewed, 
representing the universe. 


SCRC Staff Interviews 


SEDL Staff Interviews 


State Policymaker 
Interviews (Pre-Program, 
Post-Program, Post-Session) 


Up to twenty policymakers in CR communities who indicate a 
willingness to participate in study circles and up to six policymakers 
who indicate an interest in receiving reports on the program will be 
selected. 


State Policymaker 
Interviews (Follow-Up) 


A subset of the policymakers who participated in pre-program 
interviews will be selected based on responses to previous surveys and 
interviews. Selection will be based on a variety of indicators, 
including diversity, participation level in the study circle process, 
and expected use of information gained from study circles. 


Expert Participant 
Interviews (Post-Program 
and Follow-Up) 


Members of the expert and non-expert public who participated in 
study circles containing a state policymaker and who volunteer to be 
interviewed by researchers will be selected. Initial selection will be 
based on the level of participation by the policymaker and 
geographic diversity. The refined Scimple for follow-up interviews 
will be based on a variety of indicators, including diversity, self- 
assessed civic participation, interest level in the study circle process, 
and personal exposure to education experience and information. 


Non-expert Participant 
Interviews (Post-Program 
and Follow-Up) 


State Policymaker Surveys 
(Pre-Program and Follow- 
Up) 


All legislators that represent any of the CR communities in Arkansas 
and Oklahoma will receive surveys. Final respondent pool will be 
determined by response rate of this mail survey. 


Participant Survey (Pre- and 
Post-Program, administered 
before study circle session one 
and after session four) 


All study circle participants (policymaker and expert/ non-expert 
public) will receive surveys. Final respondent pool will be 
determined by response rate of this survey. 


Participant Survey (In- 
Program, administered after 
study circle session three) 


All participants in study circles that contain a policymaker 
participant will receive surveys. Final respondent pool will be 
determined by response rate of this survey. 


Observation 


The sample of study circles to be observed will be based on whether a 
state policymaker is present in the study circle. Selection among 
multiple observation opportunities will be based primarily on 
convenience, however, an attempt will be made to choose three 
geographically distinct sites in each state. 



For information on sample sizes for each procedure, see Table 4.5. 



Data Collection Timeline 



As already stated, the time period for this research study begins June 1998 and 
extends to the fall of 1999 (the Calling the Roll program, however, terminates 
with the last study circle activities in November 1998). 

Data collection activities are separated into five distinct time periods: 

1 . The pre-program period refers to the time period before study circle sessions 
begin in Arkansas and Oklahoma. 

2. The in-program period will be considered to be the time between the start of 
the first study circle session and the end of the last session for each 
community; each community will have a distinct in-program time period due 
to the variety of start and end dates among CR communities. 

3 . The post-program period also will be distinct for each community; it begins 
immediately after the end of the last CR study circle session in the 
community and continues until the beginning of the 1999 legislative session in 
that state (January 11 in Arkansas, February 1 in Oklahoma). 

4. The post-session period will be distinct for each state; it begins immediately 
with the close of the legislative session in Arkansas (March 11) and ends 
soon after the close of the Oklahoma session (May 28). 

5. The follow-up period begins three months after the end of the legislative 
session in each state and continues through August 1999, when data 
collection for this research will be complete. 

Although a more comprehensive graphic depicting data collection procedures 
in a timeline format appears on page 29, Table 4.5 briefly represents the data 
collection goals for the research study. 
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Table 4.5: Number of Interviews, Surveys, and Observations to be Conducted for 
SEDL's Calling the Roll Research Study 

Pre-Program In-Program Post-Program Post Session Follow Up 
PROCEDURE (total) before 9/98 9/98 to 12/98 10/98 to 2/99 3/99 to 6/99 6/99 to 8/99 



Coordinator Interviews (10) 


4 


2 


2 


2 


0 


SCRC staff interviews (8) 


2 


2 


2 


2 


0 


SEDL staff interviews (4) 


1 


1 


1 


1 


0 


Policymaker interviews (88) 


26 


0 


26 


26 


10 


Expert part, interviews (30) 


0 


0 


20 


0 


10 


Non-expert part, interviews (30) 


0 


0 


20 


0 


10 


Surveys of policymakers (48) 


24" 


0 


0 


0 


24 


Surveys of participants (2200) 


0 


2200" 


0 


0 


0 


Observation of study circles (6) 


0 


6 


0 


0 


0 



* Assumes a 30% response rate, N=+/-80. 

Includes 1,000 surveys administered before the first study circle session, 200 after the third study circle 
session, and 1,000 after the last study circle session. Note that SEDL is part of a shared, multi-agency effort 
to develop and implement these surveys to participants (see page 23 for details). 
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Track political/educational context 
Track media coverage of events and 
study circles project 



Data Analysis Timeline and Techniques 



Data analysis will begin with the post-program data collection period and 
continues through the fall of 1999, after final data is collected in August. Data 
analysis techniques have been selected to help develop a dependable, credible, 
and accessible accounting of both the impact of the program on the state 
education policymaking process and the factors involved in successfully 
implementing and including state policymakers in a state-wide program of 
study circles on education. 

Qualitative data analysis . Three types of coding methods suggested by Strauss 
and Corbin (1990) will be used to analyze the qualitative data (interviews, 
observation, and document review). For the sake of clarity in description, the 
analysis will be described as occurring in discrete stages; however, in reality, 
the analysis of the data will not occur in a linea’" fashion but will move back 
and forth among the different coding methods throughout the process of 
analysis. 

In the first stage of analysis, interview transcripts and secondary sources of 
data will be analyzed using open-coding methods. The data will be examined 
line by line and broken into discrete events. Conceptual labels will be attached 
to these events or happenings. Next, concepts will be grouped into thematic 
categories and the properties and dimensions of each of the categories will be 
explored. Additional data will be collected when relevant to adding detail to 
category properties and dimensions. 

In the second stage of analysis, data will be analyzed using axial coding 
methods.*^ This phase will emphasize elaboration of the themes and sub- 
themes previously identified in this section (pp. 15-20) as well as exploration 
of newly emergent themes and categories, and exceptions to both the themes 
and the emergent categories will be sought through selective (purposeful) 
sampling methods to dimensionalize the findings. The conditions that gave 
rise to various phenomena identified by open coding, the context in which 
these phenomena occur, the strategies used by others to initiate or manage 
the phenomena, and the consequences of those strategies also will be 
examined during this stage. New data will continue to be collected to validate 
and to further elucidate the dimensions. 

In the third stage, selective coding methods will be used to explicate the 
central or core themes present in the data, termed the "story-line" by Strauss 
and Corbin (1990). Also as recommended by Strauss and Corbin (1990), 
"process" or movement that is present in the data will be analyzed by 
identifying why and how actions and interactions changed, stayed the same. 



For a description of the axial coding method, see Strauss and Corbin, 1990. 
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or regressed; why events progressed in the face of changing conditions; and 
what the consequences of these events entailed. 

Finally, utilizing "member-check" procedures, experts in dialogue and select 
participants will be asked to review the findings of the coding processes 
several times during the course of the analysis and again at the end of the 
analysis to confirm the face validity of the findings. 

Quantitative data . Basic frequency and central tendency calculations will be 
used to analyze the numerical survey data. 



Monitoring and Internal Quality Control 

Given the size and complexity of the program, it will be important for SEDL 
staff to assess the quality and effectiveness of its research activities on a 
regular basis. Following are methods that will be used to ensure the quality 
and appropriateness of the research. 

Investigator Bias. In qualitative studies of this type, the investigator functions 
as the primary analytic tool. Thus, it is important to be aware that each 
individual will bring his or her unique background to the program, along 
with a certain amount of interpretive bias. Program staff will attempt to 
minimize the effect of personal bias by providing instruction to interviewers 
and observers hired to collect data. Instruction will include sensitivity to the 
topics under study, information about the scope and goals of the research, and 
discussion of bias in qualitative studies. 

Internal Audits. Periodic assessments will be made to gauge the degree to 
which the research design, activities, and products are consistent with the 
research goals. The audit will assess both the degree to which the needed data 
is being collected and the quality of the data. Changes in data collection 
methods will be made at this time if indicated. 

External Peer Review. As has been done in the initial development of this 
research study plan, informal and formal consultations will be held with 
researchers involved in similar programs to inform changes in the research 
design as the program progresses. Potential sources for these ongoing 
consultations include the SCRC, the Congressional Exchange, the Annenberg 
Foundation, the Hogg Foundation, and the University of Texas. 

Instrument Development and Testing. Interviews and surveys will be piloted 
with a select number of participants to solicit the individuals' comfort with 
the interview or survey and to recommend necessary changes before the 
instruments are implemented program wide. 



Confirmation of Reliability of Findings. As described above, SEDL's data 
gathering and analysis procedures utilize the credibility and confirmability 
criteria recommended by Lincoln and Cuba (1985) in order to establish the 
reliability of research findings. Specific techniques include: 

• Triangulation of data. This technique uses several types of data, namely 
interviews, observations, surveys, and document review. 

• Member checks (external). Individuals involved in the program and experts 
in the field are asked to comment on the face validity of the findings. 

• Member checks (internal). SEDL researchers will monitor and perform cross 
checks of each other's coding and analysis of the data. 

• The constant comparative method recommended by Strauss and Corbin 
(1990). Data are analyzed as collected, and emerging categories and themes 
are constantly compared back to the data and explored in remaining data 
collection activities. 

• Referential adequacy. Portions of the data are not included in the initial data 
analysis but are reserved for comparison to the models that emerge from the 
data. 

Recruitment 



Interview Recruitment. SEDL staff will have primary responsibility for the 
recruitment of state policymakers and community members for the 
interviews planned for this research. SEDL will not only recruit policymakers 
for interviews but also will encourage their participation in the study circle 
program in both states. As SEDL staff work with program coordinators to 
solicit the participation of policymakers in the program through letters and 
personal communication, policymakers will also be informed and invited to 
become key information sources for the research study. SEDL staff will rely on 
assistance from local coordinators and facilitators to recruit the non-expert 
and expert public for participation in interviews, as well as for cooperation in 
completing survey forms as discussed below. 

Survey Recruitment. Surveys of program participants (policymakers and 
constituents) will be integrated into the study circle implementation process 
with the help of program partners and local coordinators. SEDL researchers 
will inform program facilitators and local organizers about the survey 
component of Calling the Roll. Detailed instructions regarding survey 
administration will be provided to all facilitators, clearly stating when 
surveys should be provided to participants and how many surveys to 
conduct. Facilitators will be requested to assist those participants who have 
low reading levels in completing surveys. Facilitators also will ask 
participants to indicate their willingness to volunteer to receive further 
communication from SEDL researchers regarding follow-up interviews. 
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Those volunteers who are selected for post-program and follow-up 
interviews will be contacted for recruitment via phone and mail by SEDL 
staff. 




Confidentiality and Informed Consent 

Related to the issue of recruitment are the consent procedures to be 
implemented for this study. Participation in research activities by state 
policymakers and the expert and non-expert public will be completely 
voluntary and will in no way affect their right to otherwise participate in the 
CR study circle program. Consent will be received from study subjects at the 
initial invitation to participate, before actual interviews or surveys are 
conducted, and before interviews are audiotaped or otherwise recorded. Data 
recorded as a result of surveys or interviews will be kept confidential, unless 
the interview subject gives written permission to SEDL to quote or otherwise 
reproduce identifiable information. Otherwise, data results will be 
disseminated only in the aggregate, as anonymous statements, or as 
unidentifiable case studies. Personal contact information will be asked of all 
research subjects in order to facilitate follow-up surveys and interviews; 
however, this information will be kept confidential, and participants will be 
contacted only if prior permission is granted. Written notice of SEDL's 
intentions regarding the use of all data collected from research subjects will be 
provided to all who participate. 

Instrumentation 



Data collection instruments will undergo review by Calling the Roll partners 
as part of SEDL's internal and external peer review process. The questions and 
themes that are presented at the beginning of this section will provide the 
initial basis for specific questions included in instruments. After comments 
are received by program partners, SEDL staff will pilot the draft instruments 
for effectiveness and ease of use before final documents are created. Baseline 
data collected using these instruments will go on to inform the next round of 
interviews and surveys. Post-program and follow-up instrumentation will be 
developed following a similar process. 

Dissemination of Results 



One of the primary tasks of SEDL is to develop a mix of products that will 
inform state and local policy audiences about the usefulness and feasibility of 
deliberative dialogue processes and how the particular method tested — the 
study circle process — could be implemented in their communities or states. 
Products will include: report(s) discussing both process and impact results of 
the SEDL research study; an informative video about the Calling the Roll 
program that tells the "Whole Story" of the Arkansas and Oklahoma 
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experiences; and information for policymakers in the form of policy briefs. If 
research results warrant, SEDL also will produce a guidebook that provides 
assistance to state policymakers and groups interested in replicating the 
program in other locations — particularly for the purpose of including state 
policymakers and contributing to the state education policymaking process. 
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SECTION V 



RESEARCH DESIGN THAT CAPTURES THE WHOLE STORY: 
COLLABORATIVE STUDY OF STATE-WIDE PROGRAMS OF 
STUDY CIRCLES ON EDUCATION 



Summary 

Each of the four Calling the Roll partners will conduct a separate research 
study based on its specific interests in participating in the CR program. The 
potential result is an interrelated and mutually supporting set of studies that 
"captures the whole story" of the 1998 state-wide programs of study circles on 
education in Arkansas and Oklahoma. This research will collect information 
about the local and state processes of deliberation and coordination and also 
will describe the impact each has on participants — including community 
members, their elected representatives, and the organizers of state-wide 
democratic deliberations. 

SEDL's policy staff will specifically focus on the impact of study circles on state 
education policymaking, and the process of implementing a state-wide 
program of study circles on education that involves policymakers. This 
section presents a brief discussion of the foci of the other three partners' 
research, the data collection of another SEDL project (Language and Diversity 
Program), and opportunities for collaboration among all research teams. 



Foci of Partners' Research Studies 



Arkansas Friends for Better Schools are contracting with Dr. Larry Dickerson 
(Center for the Research of Teaching and Learning, University of Arkansas at 
Little Rock) to develop and conduct an evaluation of the local impact of study 
circles in participating Arkansas communities. Data collection includes 
descriptive statistics on what happens in study circle communities as a result 
of the program — with an emphasis on quantitative evidence in four areas: 
citizen participation, study circle program implementation, study circle 
program institutionalization, and study circle impact on the community. 

The Oklahoma League of Women Voters is contracting with Dr. Wil Scott 
(Department of Sociology, University of Oklahoma at Norman) to assess 
positive change along broad measures of citizen involvement. Although this 
includes an interest in hard data on study circle participants, deeper interest is 
in how the study circle process works and what its effects are on participants 
as community members and citizens In-depth observation of a few study 




43 



35 



circlGS, which C3n yield im.port3nt findings shout psrticipsnt intersetion snd 
the content of the dialogue on education, will be the focus of this study. 

SCRC has greatest interest in conducting an implementation evaluation. The 
Calling the Roll program in Arkansas and Oklahoma provides an 
opportunity to learn how the process of organizing on a state level compares 
with organizing at the community level. As such, the SCRC evaluation 
includes data collection from local coordinators that complements 
information collected by SEDL from the state coordinators. SCRC staff also are 
interested in validating a study circle model that includes federal-level 
policymakers in community-wide study circle program (recently developed by 
Congressional Exchange). 

SEDL's language and cultural diversity staff members would like to learn how 
the study circle process works in groups of people with diverse backgrounds. 
They propose data collection limited to observation and participant 
interviews in up to ten study circles during the fall 1998 program. Although 
the nature of the work does not permit the kind of cooperation possible 
among the other four studies, SEDL diversity staff activities and products, as a 
result of this research, will be shared with all partners to benefit future study 
circle programs in communities of diverse languages and cultures. 



Collaboration 

Because there are a number of shared research interests among the partners, 
the potential for developing a unified research plan is strong. The benefits of 
a collaborative research effort are clear: collaboration would expand both the 
scope and the value of CR research beyond that which could be achieved by a 
single partner acting alone. A collaborative research effort would require at 
the very least the development of a complementary or unified research 
design, the use of complementary methods and tools, and the designation of a 
research coordinator whose role (at a minimum) is to coordinate regular 
communications among CR partners. The coordinator will also facilitate the 
data sharing among partners that has been formally agreed upon in the 
Arkansas and Oklahoma subcontracts with SEDL. 

CR partners will use parallel protocols and create shared data collection 
instruments. Program researchers will develop a common participant survey 
instrument for use by all of the partners. Dr. Dickerson will oversee data entry 
and scanning of survey data, and the partners will work together to 
administer surveys to all study circle participants. In its updated facilitators' 
guidebook, SCRC has provided an observation checklist that will t)e adapted 
to record content information from study circle sessions that is of interest to 
SEDL and UON. Evaluators will communicate regularly via teleconferencing 
and email to consult on design issues and development of survey forms. 
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interview protocols, databases, and dissemination opportunities. This 
communication also will allow researchers to identify further opportunities 
to complemiGnt and enhance each other's work. 



Table 5.1: Primary Research Foci of Partners 



Research Focus 

Implementation of education study circles (SCs) 
Implementation of SCs that include state 
policymakers 

Implementation of state-wide SC programs 

Degree to which SCs facilitate presentation of 
multiple viewpoints 

Impact of SCs on policymaker-community 
relationship 

Impact of SCs on information available to state 
policymakers 

Impact of SCs on policymakers' and/ or 
community's level of civic involvement 

Impact of SCs on policymakers' and/ or 
community members' knowledge of education 
issues 



SEDL 

X 

X 

X 

X 

X 

X 

X 

X 



Impact of SCs on educators 
Impact of SCs on state policymaking process 
Long-term structural impact of SCs 
Impact of SCs on local education system 



X 

X 

X 



SCRC 

X 

X 



X 



AFBS 

X 

X 

X 



X 



X 

X 




LWVO 

X 

X 

X 

X 

X 
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GLOSSARY OF TERMS 



Arkansas Friends for Better Schools — Arkansas Friends for Better Schools is 
an alliance of advocates for public education representing education, business, 
civic, and religious organizations. Sponsored by the Arkansas Interfaith 
Conference, Arkansas Friends was founded in late 1993 and funded by the 
Winthrop Rockefeller Foundation through March 1997. Friends also enjoys 
substantial in-kind support from Southwestern Bell Telephone. Arkansas 
Friends was originally founded to provide information to its members and to 
the public about systemic school reform, particularly as defined under Act 236 
of 1991. A key interest of the alliance is to build support for reform and public 
schools in the state. 

action forum — A community-wide event at the conclusion of the study circle 
program. The action forum can inform participants about future action 
opportunities or provide the opportunity for groups and individuals to form 
new coalitions and plan more study circles. This forum is also a chance for 
participants to share experiences with each other and officials or 
policymakers. 

Calling the Roll — A collaborative program that will implement state-wide 
study circles on education in Arkansas and Oklahoma during September 
through November 1998. 

deliberative dialogue — A mode of communication that enables individuals 
to constructively discuss an issue of shared concern with the purpose of 
increasing understanding of diverse perspectives and coming to a common 
sense of direction and potential action. 

expert public — Members of the local community who have expertise or 
specific knowledge regarding the topic discussed in a study circle. For Calling 
the Roll, expert public members will include school administrators, 
educators, education advocates, and education researchers. 

facilitator — The designated discussion leader for a study circle session. 
Facilitators are not necessarily experts in the topic of discussion but do receive 
training in helping people listen and engage in constructive dialogue. 

focus group — Structured public gathering in which the sponsoring entity 
solicits the opinions of participants on a single issue. Generally used as a 
method to extract opinions from a specially chosen group of participants who 
are screened using established criteria. 

issues forums — A model that puts into practice concepts of deliberative 
dialogue. Issues forums are similar to study circles; however, issues forums 
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are planned as large group discussions that occur in a single event, whereas 
study circles emphasize a series of small group dialogues over time. 

kick-off forum — A community-wide event that initiates the study circle 
program. The kick off helps draw media and community visibility to the 
study circles and helps recruit potential sponsoring organizations. 

League of Women Voters of Oklahoma — A nonpartisan political 
organization that encourages the informed and active participation of citizens 
in government and influences public policy through education and advocacy. 

non-expert public — Members of the local community who have no special 
knowledge or expertise regarding the study circle topic. 

public engagement — An interactive process that provides an opportunity for 
the public to participate in dialogue pertaining to decisions that will impact 
community structures and systems. 

study circle process — ^A specific approach to broad-scale engagement of 
community members in small group deliberation. 

Study Circles Resource Center (SCRC) — A project of Topsfield Foundation to 
promote the study circle model. SCRC provides technical assistance, 
materials development, and research to help states and communities 
implement their own study circles. Congressional Exchange is a similar 
Topsfield project that applies the model at the national level. 

state coordinator — Designated individual who is responsible for the state- 
level coordination of Calling the Roll in each of the two states. 

state policymakers — Policymakers and state-level decisionmakers. 

state-wide study circle program — A program coordinating the 

implementation of community-wide study circles in multiple communities 
across a state. 
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LIST OF ACRONYMS 



AFBS — Arkansas Friends for Better Schools 
CR — Calling the Roll 

LWVO — League of Women Voters of Oklahoma 
OERI — Office of Educational Research and Improvement 
SCs — study circles 

SCRC — Study Circles Resource Center, a project of the Topsfield Foundation, 
Inc. 

SEDL — Southwest Educational Development Laboratory 

UALR — University of Arkansas at Little Rock, Center for the Research of 
Teaching and Learning 

UON — University of Oklahoma, Norman, Department of Sociology 
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